Evolving Inborn Knowledge For Fast Adaptation in Dynamic POMDP Problems

Rapid online adaptation to changing tasks is an important problem in machine learning and, recently, a focus of meta-reinforcement learning. However, reinforcement learning (RL) algorithms struggle in POMDP environments because the state of the system, essential in a RL framework, is not always visible. Additionally, hand-designed meta-RL architectures may not include suitable computational structures for specific learning problems. The evolution of online learning mechanisms, on the contrary, has the ability to incorporate learning strategies into an agent that can (i) evolve memory when required and (ii) optimize adaptation speed to specific online learning problems. In this paper, we exploit the highly adaptive nature of neuromodulated neural networks to evolve a controller that uses the latent space of an autoencoder in a POMDP. The analysis of the evolved networks reveals the ability of the proposed algorithm to acquire inborn knowledge in a variety of aspects such as the detection of cues that reveal implicit rewards, and the ability to evolve location neurons that help with navigation. The integration of inborn knowledge and online plasticity enabled fast adaptation and better performance in comparison to some non-evolutionary meta-reinforcement learning algorithms. The algorithm proved also to succeed in the 3D gaming environment Malmo Minecraft.Comment: 9 pages. Accepted as a full paper in the Genetic and Evolutionary Computation Conference (GECCO 2020

Evolutionary bits’n’spikes

We describe a model and implementation of evolutionary spiking neurons for embedded microcontrollers with few bytes of memory and very low power consumption. The approach is tested with an autonomous microrobot of less than 1 in 3 that evolves the ability to move in a small maze without human intervention and external computers. Considering the very large diffusion, small size, and low cost of embedded microcontrollers, the approach described here could find its way in several intelligent devices with sensors and/or actuators, as well as in smart credit cards. Artificial Spiking Circuits Most biological neurons communicate by sending pulses across connections to other neurons. The pulse is also known as “spike ” to indicate its short and transient nature. Neurons are affected by incoming spikes and generate a spike when their membrane potential becomes larger than a threshold. Spike generation is followed by a short “refractory period ” during which the neuron cannot generate another spike. Computational models of spiking neurons are attracting increasing interest in engineering and computer science (Maas & Bishop 1999). On the one hand, computer simulations of spiking networks can help to address specific questions in neuroscience, such as how biological neurons communicate with each other (Koenig, Engel, & Singer 1996; Rieke et al. 1997). On the other hand, a better understanding of spiking neurons is leading to the development of new neuromorphic devices (Horiuchi 2001), some of which may replace lesioned fibers or sensory organs. In addition, we argue that networks of spiking neurons represent suitable control systems for autonomous behavioral systems, 1 such as situated autonomous robots, because temporal patterns of sensory-motor events may be captured and exploited more efficiently (i.e., with fewer neurons or with higher probability) by the intrinsic time-dependent dynamics of spiking neurons than by 1 They certainly showed to be excellent control systems for biological organisms! other connectionist models (Rumelhart, McClelland, &amp